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ABSTRACT 

The  Agricultural  Resource  Management  Study  (ARMS)  Phase  III  collects  specific  information  from  farm 
operators  regarding  their  rates  of  production,  expenses,  returns,  and  profits.  An  ARMS  panel  plus  pilot 
study  was  started  in  survey  year  1998  and  continued  in  survey  year  1999  to  assess  operator  cooperation 
rates  over  a  three  year  period.  First  year  results  can  be  found  in  RD-99-05.  During  the  second  year,  a 
new  sample  was  added  and  operators  from  the  first  year  were  re-contacted  to  complete  another  ARMS 
questionnaire. 

For  the  pilot  study,  we  increased  the  public  relations  and  promotion  activities  for  a  small  group  of 
operators  and  enumerators  in  order  to  retain  operator  cooperation  over  a  period  of  three  years.  In 
general,  even  with  an  intensive  promotion  effort  up  front,  we  obtained  a  slightly  lower  response  rate  on 
the  ARMS  panel  than  on  the  regular  ARMS  survey. 
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SUMMARY 


Phase  III  of  the  Agricultural  Resource  Management  Study  (ARMS)  is  currently  conducted 
annually  with  a  new  sample  of  farm  operators  selected  each  year.  A  new  longitudinal  panel 
survey  methodology  is  under  discussion.  The  proposed  “Panel  Plus”  design  would  collect 
detailed  economic  information  from  a  small  panel  of  operators  over  a  period  of  three  years,  while 
general  economic  information  would  be  collected  for  a  larger  “plus”  sample  of  operators  that 
changes  every  year. 

One  of  the  components  necessary  for  this  design  is  long  term  farm  operator  cooperation  for  the 
panel  sample.  To  assess  the  feasibility  of  a  three  year  participation  commitment  from  operators, 
a  panel  plus  pilot  study  was  initiated  in  survey  year  1998  (data  collection  in  February  1999)  in 
Virginia  and  Missouri.  A  variety  of  new  promotional  ideas  was  used  to  obtain  and  retain 
cooperation  over  the  long  term. 

Response  rates  on  the  panel  plus  pilot  sample  were  lower  than  on  the  regular  ARMS  sample  - 
completed  by  the  same  enumerators,  indicating  that  the  three  year  aspect  of  the  study  may  have 
led  to  a  decrease  in  response  rates  in  the  initial  year.  In  general,  nonpanel  response  rates  for  the 
enumerators  working  on  the  pilot  were  higher  in  the  initial  pilot  year  than  rates  for  the  past 
survey  year.  This  could  be  attributed  to  the  overall  increased  public  relations  and  promotion  that 
was  directed  towards  enumerators. 

Even  with  additional  public  relations  and  promotion  efforts,  response  rates  declined  further  in  the 
second  year,  but  we  achieved  a  retention  rate  of  about  88  percent  in  both  states. 
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INTRODUCTION 

Annually,  the  National  Agricultural 
Statistics  Service  (NASS)  and  the  Economic 
Research  Service  (ERS)  conduct  the 
Agricultural  Resource  Management  Study 
(ARMS),  a  survey  collecting  financial 
information  and  farming  practices  from  farm 
operations  across  the  United  States.  The 
average  interview  length  is  90  minutes,  and 
many  data  items  are  perceived  by  NASS  as 
sensitive  and/or  difficult  for  farm  operators 
to  answer. 

Approximately  17,000  farm 
operations  across  the  country  are  selected  for 
the  ARMS  data  collection  process  each  year. 
In  recent  years,  overall  national  response 
rates  for  this  study  have  been  about  60 
percent. 

In  an  attempt  to  decrease  overall 
respondent  burden  and  increase  response 
rates,  NASS  and  ERS  are  developing  and 
evaluating  a  proposal  that  involves  obtaining 
detailed  information  from  a  relatively  small 
panel  of  producers  across  several 
consecutive  years,  while  collecting  limited 
information  from  a  larger  group  of 
producers. 

This  “panel  plus”  data  collection 
methodology  would  involve  selection  of  two 
ARMS  samples  for  data  collection.  The 
first  sample  would  be  called  the  “panel”  and 
would  include  approximately  3,500 
operators  who  would  annually  provide 
detailed  economic  and  management 
information  for  their  operations  over  three 
consecutive  years.  The  second  sample 
would  be  called  the  “plus”  and  would 
include  approximately  10,000  operations 
who  would  provide  limited  economic  data 
needed  for  regional  and  national  level 
estimation.  The  two  samples  would  be 
linked  using  appropriate  modeling 


techniques  to  produce  estimates  which 
would  support  NASS  and  ERS  data  needs. 

METHODOLOGY 

Before  implementing  the  panel  plus 
methodology  for  the  ARMS,  a  small  panel 
plus  pilot  study  was  started  in  two  states  to 
look  at  ways  to  increase  current  response 
rates,  track  response  and  retention  rates, 
investigate  farm  operator  reaction  to  a  three 
year  study,  and  to  examine  alternative  data 
collection  options  that  may  make  reporting 
easier  and  less  time  consuming  in  the  future. 

This  report  only  covers  the  public 
relations  and  promotion  aspect  of  the  pilot 
study.  Development  of  a  shorter 
questionnaire  and  the  panel  plus  sampling 
and  modeling  will  be  addressed  elsewhere. 

Data  collection  for  the  pilot  study 
started  in  February  1999,  collecting  data  for 
the  1998  calendar  year,  and  continued  with 
the  second  year  data  collection  starting  in 
February  2000  for  the  1999  survey  year.  For 
the  rest  of  this  report,  year  references  w  ill  be 
to  the  survey  year,  not  when  the  collection 
occurred.  So,  1997  refers  to  survey  year 
1997,  1998  refers  to  survey  year  1998,  and 
1999  refers  to  survey  year  1999. 

The  first  cohort  of  operations  in  the 
panel  plus  pilot  study  was  initially 
interviewed  for  the  panel  in  1998  and  re¬ 
interviewed  in  1999.  The  first  cohort 
involved  a  sample  of  96  total  farm 
operations  in  two  states,  Missouri  (MO)  and 
Virginia  (VA).  In  addition,  a  second  cohort 
of  59  farm  operations  was  added  in  survey 
year  1999. 

First,  enumerators  were  selected  to 
work  on  the  project  based  on  their  past 
performance.  For  the  1998  survey  year, 
enumerators  with  traditionally  the  highest 
response  rates  were  selected.  In  general,  for 
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the  1999  survey  year,  enumerators  with 
more  average  response  rates  were  selected. 
This  non-randomized  selection  was  used 
because  during  the  first  year  of  the  study,  we 
wanted  to  be  sure  that  the  new  procedures 
would  work  for  the  best  enumerators  before 
expanding  to  enumerators  with  lower 
response  rates.  Also,  we  were  not  as 
concerned  with  statistical  reliability  as  we 
were  with  the  feasibility  of  doing  a  larger 
scale  study  using  these  procedures. 

Basically,  if  we  discovered  that  our  response 
rates  suffered  dramatically  with  these 
enumerators,  we  would  not  spend  the  money 
for  a  large  scale  study  using  all  enumerators. 

Secondly,  operations  that  were  in  the 
active  ARMS  sample  in  the  counties  where 
these  enumerators  worked  were  identified  as 
ARMS  panel  plus  pilot  sample.  Therefore, 
all  operations  in  the  panel  plus  pilot  study 
were  originally  part  of  the  ARMS  sample 
and  would  have  been  interviewed  for  ARMS 
regardless  of  the  pilot  study  (for  the  first 
year). 

Four  enumerators  worked  on  the 
original  Virginia  cohort  of  47  pilot 
operations.  Six  enumerators  worked  on  the 
original  Missouri  cohort  of  49  pilot 
operations.  (These  10  enumerators  will  be 
referred  to  as  the  first  group  of  enumerators.) 
During  the  second  year  of  the  pilot  study,  all 
1998  operations,  regardless  of  their  1998 
status,  were  contacted  again.  During  this 
second  year  contact,  enumerators  verified 
out-of-business  status,  returned  to  interview 
operations  who  either  completed  the 
interview  or  were  inaccessible,  or  attempted 
to  convert  refusals. 

For  the  1999  survey  year,  we  added 
enumerators  and  samples  in  both  states.  A 
total  of  30  operations  were  added  in  VA  (17 
were  attempted  by  3  new  enumerators,  while 
13  were  attempted  by  two  of  the  panel 


enumerators  from  1998).  Four  new 
enumerators  contacted  29  operations  in  MO 
(These  7  enumerators  will  be  referred  to  as 
the  second  group  of  enumerators). 

The  Census  of  Agriculture  was 
conducted  in  January  -  May  1998,  so  no 
pilot  study  data  were  collected  during  the 
same  year  as  the  Census. 

Because  we  did  not  randomly  select 
enumerators  and  since  we  have  such  a  small 
sample,  the  results  are  not  necessarily 
representative  of  those  we  would  see  in 
other  states  or  with  other  enumerators.  This 
small,  relatively  inexpensive  pilot  study  was 
done,  in  part,  to  determine  the  feasibility  of 
doing  a  larger,  more  comprehensive  test  of  a 
longitudinal  design. 

PUBLIC  RELATIONS  AND  PROMOTION 

In  order  to  gain  cooperation  for  three 
years,  one  of  the  main  objectives  of  the 
study  was  to  build  rapport  between 
enumerators,  NASS  personnel,  and 
respondents.  For  details  on  the  specific 
methodology,  refer  to  research  report  RD- 
99-05  entitled  “ARMS  Panel  plus  Pilot 
Study:  Results  from  the  1998-1999  ARMS 
Panel  Plus  Pilot  Study”  (Ott,  1999). 

Research  report  RD-99-05  gives  detailed 
methodology,  findings  and  results  from  the 
first  year  of  the  pilot  study.  This  report 
(RDD-00-06)  gives  an  overview  of  the 
methodology  and  results  from  the  second 
year. 

The  basic  survey  implemention 
methodology  included  the  following:  pilot 
operations  were  sent  an  additional  pre¬ 
survey  letter  from  the  state  department  of 
agriculture,  and  were  given  incentives  when 
the  enumerators  set  up  the  interviews  or 
during  the  interviews. 

In  addition,  a  two  person  team 
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consisting  of  the  enumerator  and  a 
headquarter’s  or  state  office  person  visited 
the  operation  for  the  interview.  Pilot 
enumerators  were  given  incentives  and 
attended  special  training  sessions. 
Depending  on  the  year,  the  enumerator 
incentives  included  hand  bags,  t-shirts,  or 
polo  shirts.  All  of  these  procedures  were 
used  for  both  years  of  the  pilot  study  during 
the  first  year  the  operation  was  contacted. 

There  were  post  interview  contacts 
with  both  cohorts  of  pilot  farm  operations  as 
well.  A  thank  you  card  was  immediately 
sent  by  the  state  office  or  the  enumerator  to 
each  respondent  and  a  refusal  letter  sent  to 
each  refusing  operation.  The  refusal  letter 
acknowledged  the  operation’s  refusal  to 
participate  in  the  ARMS  and  gave  some 
information  about  how  the  data  are  used. 

Between  interview  contacts,  the 
enumerator  made  a  special  visit  to  each 
operation.  They  delivered  an  Individualized 
Farm  Analysis  (for  cooperators  only) 
comparing  their  operation  to  others  in  their 
region,  along  with  a  tin  of  cookies  to  each 
respondent  and  most  refusals  in  the  fall. 

They  did  not  collect  any  data  during  this 
contact  and  reiterated  the  offer  of  state  office 
assistance  with  other  USDA  or  farm  related 
issues  first  offered  during  the  earlier 
interview  contact. 

During  the  second  year  of  data 
collection  for  the  first  cohort  of  operations, 
panel  operators  were  sent  a  presurvey  letter 
and  were  contacted  by  the  same  enumerator. 
The  interview  was  done  by  the  enumerator 
alone,  as  there  was  no  two-person  team  used 
for  the  second  interviews. 


RESULTS 
First  year  cohort 

Even  though  we  asked  respondents 
to  be  in  the  study  for  three  consecutive 
years,  we  hoped  to  see  first  year  response 
rates  on  the  pilot  study  that  were  equal  or 
higher  than  response  rates  on  the  rest  of  the 
operational  ARMS  sample  due  to  the 
increased  personal  attention  we  gave  farm 
operators.  We  also  hoped  to  see  overall 
response  rates  for  1998  increase  compared  to 
1997  because  we  had  increased  the  amount 
of  attention  and  importance  placed  on  the 
survey,  and  relayed  that  to  enumerators.  In 
addition,  because  of  the  increased  focus  on 
data  uses  and  the  enumerator  visits  in  the 
fall  (delivering  the  farm  analysis  and/or 
cookie  tin),  we  hoped  to  achieve  a  high 
retention  rate  for  the  1999  panel. 

Table  1  shows  the  1997,  1998.  and 
1999  response  and  refusal  rates  obtained  by 
just  the  first  group  of  enumerators 
(enumerators  who  worked  on  both  years  of 
the  pilot  study  in  each  state).  These  rates 
and  the  corresponding  numbers  of  total 
cases  are  also  shown  graphically  in 
Appendix  B.  The  first  year,  there  was  a  total 
of  15  panel  refusals,  8  in  Missouri  and  7  in 
Virginia.  The  second  year,  there  was  a  total 
of  20  panel  refusals,  10  in  each  state. 

Rates  are  divided  into  five  categories 
based  on  the  year  they  were  interviewed  and 
whether  they  were  included  in  the  panel 
portion  of  the  survey.  In  Table  1,  “1997 
Nonpanel”,  “1998  Nonpanel”,  and  “1999 
Nonpanel"  include  the  cases  that  were  not 
included  in  the  panel,  but  were  interviewed 
by  the  panel  enumerators.  Keep  in  mind  that 
all  interviews  were  the  traditional  ARMS 
interview  (90  minute  average  interview 
time),  since  we  were  not  yet  testing  the  short 
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“plus”  questionnaire.  The  categories  “1998 
Panel”  and  “1999  Panel”  include  only  the 
panel  cases  for  each  year  interviewed  by  the 
panel  enumerators.  The  rates  were 
calculated  using  the  following  formulas. 

Response  Rate  = 

Completes 

Completes  +  Refusals  +  Inaccessibles 
Refusal  Rate  = 

Refusals 

Completes  +  Refusals  +  Inaccessibles 


Table  1:  First  Year  Panel  Enumerators 


1997 

1998 

1999 

Response  Rates* 

— 

-percent 

— 

Virginia 

Panel 

- 

83 

75 

Nonpanel 

91 

95 

88 

Missouri 

Panel 

- 

73 

67 

Nonpanel 

77 

86 

68 

Refusal  Rates* 

— 

-percent 

— 

Virginia 

Panel 

- 

15 

22 

Nonpanel 

6 

5 

8 

Missoun 

Panel 

- 

17 

23 

Nonpanel 

19 

14 

25 

*  Total  N’s  are  included  in  Appendix  B  and  C.  All 
differences  are  insignificant  due  to  sample  size. 


There  is  one  difference  in  the  way 
response  and  refusal  rates  are  calculated 
from  the  original  pilot  study  research  report 
(RD-99-05).  In  this  new  research  report 
(RDD-00-06),  out-of-business  operations  (2 
in  1998,  6  additional  in  1999)  are  excluded 
from  the  denominator  for  these  rates.  Out- 
of-business  reports  were  included  in  the 
denominators  in  research  report  RD-99-05. 
Partial  interviews  are  unusable  for  ARMS 
and  are  therefore  considered  refusals  in  both 
reports. 

As  illustrated  in  Table  1  above,  1998 
panel  response  rates  were  lower  than 
nonpanel  rates  each  year,  possibly  showing 
that  respondents  and/or  enumerators  reacted 
negatively  to  the  three  year  aspect  of  the 
study,  even  though  we  tried  to  emphasize  the 
positive  uses  of  the  data  and  promote  our 
Agency.  The  1998  nonpanel  response  rates 
in  both  states  were  higher  than  the  1997 
nonpanel  response  rates.  Possibly,  this  is 
because  of  our  PR  and  publicity  efforts 
directed  at  enumerators.  These  motivational 
aspects  of  the  PR  and  publicity  focused  on 
the  general  ARMS  survey  as  much  as  the 
pilot  study,  so  they  could  have  affected 
enumerators  in  all  of  their  interviewing,  not 
just  on  the  panel. 

In  1999,  both  the  nonpanel  and  the 
panel  response  rates  dropped  in  both  states 
to  levels  below  the  1998  and  1997  rates. 

We  expected  some  decrease  in  the  panel 
response  rates  due  to  the  longitudinal  nature 
of  the  study.  However,  we  did  not  expect 
such  large  decreases  in  the  nonpanel 
response  rates.  Of  course,  our  number  of 
cases  is  small  and  we  cannot,  therefore, 
place  too  much  significance  on  the  decrease. 
Also,  there  could  be  any  number  of 
explanations  for  a  decrease  including  factors 
outside  of  our  control.  These  factors  may 
include  changing  agricultural  economics. 
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public  opinion,  or  weather  conditions,  all  of 
which  could  affect  response  rates.  However, 
it  could  be  that  our  extensive  public 
relations  and  promotion  campaign  used 
during  the  1998  survey  year  “wore  off’  by 
the  1999  survey  year.  Indeed,  we  did  less 
promotion  with  these  enumerators  during  the 
1999  survey  year. 

Refusal  rates  for  the  panel  compared 
with  the  nonpanel  cases  differ  by  state.  In 
Missouri,  the  panel  refusal  rates  were  fairly 
close  to  the  refusal  rates  for  the  nonpanel 
cases  within  the  same  year.  However,  the 
panel  refusal  rates  were  much  higher  than 
nonpanel  refusal  rates  in  Virginia.  It  is 
unclear  why  this  would  be  the  case,  except 
there  were  more  inaccessible  cases 
(operations  that  cannot  be  reached  during  the 
enumeration  period)  in  Missouri.  It’s 
possible  that  these  hard  to  reach  farm 
operators  would  refuse  more  often  than 
those  that  are  easier  to  reach. 

It  is  promising  that  the  response  rate 
did  not  decrease  significantly  for  the  panel 
given  the  three  year  aspect  of  the  study.  It  is 
difficult  to  assess  how  much  of  the  response 
rate  was  related  to  the  added  burden  of 
participation  for  three  years  and  how  much 
was  related  to  the  added  public  relations, 
promotion  work,  and  special  attention  we 
gave  to  panel  enumerators  and  respondents. 
Again,  for  many  reasons,  including  the  non- 
random  selection  of  enumerators  and  the 
small  sample  sizes,  we  cannot  generalize 
these  results  to  our  full  population. 

To  compare  the  1998  and  1999 
outcomes  for  the  farm  operation  interviews 
involved  in  the  longitudinal  panel.  Table  2 
shows  the  number  of  complete,  refusal, 
inaccessible,  and  out-of-business  (OOB) 
cases  for  both  years  of  data  collection. 


Table  2:  Status  of  Panel  Cases 


Status 

1998  Panel 

1999  Panel 

Complete 

73 

63* 

Refusal 

15 

20 

Inaccessible 

(noncontact) 

6 

6 

Out-of- 

Business 

2 

6 

Total 

96 

95* 

*one  1998  complete  panel  case  was  removed  from  the 
panel  for  subsequent  years 

There  were  19  cases  that  changed 

status  between  the  two  years. 

Table  3  shows 

these  changes. 

Table  3. 

Changes  in  Status 

Status  Change 

Number 

Complete  to  Refusal  9 


Complete  to  Refusal  9 

Complete  to  Inaccessible  2 

Complete  to  OOB  1 

Refusal  to  Complete  2 

Refusal  to  Inaccessible  1 

Refusal  to  OOB  1 

Inaccessible  to  Complete  1 

Inaccessible  to  OOB  2 


The  second  year  retention  rate  was 
high  in  both  states.  Missouri’s  retention  rate 
was  89  percent,  with  31  of  the  35  complete 
reports  from  1998  participating  in  1999.  and 
Virginia’s  retention  rate  was  86  percent  with 
32  of  the  37  participating  in  1999  (one 
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operation  was  removed  from  the  panel 
because  of  the  large  size  of  the  operation  and 
its  potential  impact  on  the  floriculture 
survey). 

In  summary,  although  our  results 
cannot  be  generalized  to  all  enumerators  or 
respondents,  we  do  see  lower  response  rates 
on  the  panel  portion  of  the  ARMS  survey 
than  on  the  regular  ARMS  survey. 

Nonpanel  rates  increased  during  the  year 
that  we  increased  the  PR  and  promotion 
activities  with  enumerators  and  respondents, 
and  decreased  the  following  year.  The 
promotional  activities  geared  towards 
enumerators  and  respondents  may  be 
effective  in  the  short  term,  but  may  wear  off 
over  the  year. 

Second  year  cohort 

As  stated  earlier,  a  second  group  of 
enumerators  interviewed  a  new  cohort  of 
operations  that  were  added  for  the  second 
year.  The  same  public  relations  and 
promotional  activities  were  used  for  these 
enumerators  and  respondents.  Response  and 
refusal  rates  for  the  operations  interviewed 
by  the  second  group  of  enumerators  are 
shown  in  Table  4.  They  are  also  shown 
graphically  with  the  total  number  of  cases  in 
each  cell  in  Appendix  C.  In  Virginia,  two  of 
the  first  year  enumerators  were  used  to 
complete  some  of  these  interviews.  Their 
response  rates  are  not  included  in  Table  4. 
Presentation  and  discussion  of  their  response 
rates  for  the  second  year  cohort  are  in 
Appendix  A.  The  numbers  for  Virginia  in 
Table  4  are  for  just  the  second  group  of 
enumerators. 


Table  4:  Second  Year  Panel  Enumerators 


1997 

1998 

1999 

Response  Rates 

— 

-percent 

— 

Virginia 

Panel 

- 

- 

59 

Nonpanel 

- 

70 

81 

Missouri 

Panel 

- 

- 

52 

Nonpanel 

- 

75 

43 

Refusal  Rates 

— 

-percent 

— 

Virginia 

Panel 

- 

- 

41 

Nonpanel 

- 

23 

20 

Missouri 

Panel 

- 

- 

38 

Nonpanel 

- 

15 

50 

Similar  to  the  first  year  enumerator 
trend,  the  second  group  of  Virginia 
enumerators  obtained  lower  response  rates 
on  the  panel,  but  higher  nonpanel  response 
rates  for  the  1999  ARMS  than  the  1998 
ARMS.  This  is  not  the  case  for  the  second 
group  of  Missouri  enumerators.  In  fact,  the 
nonpanel  response  rate  was  much  lower  in 
1999  than  in  1998.  Also,  the  panel  response 
rate  for  these  enumerators  was  higher  than 
the  nonpanel  response  rate.  However, 
according  to  the  state  office,  the  counties 
selected  for  1999  have  been  traditionally  low 
response  rates  counties.  In  addition,  in  some 
cases,  the  enumerators  were  interviewing  in 
areas  that  were  not  in  their  regular  work 
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area.  This  makes  it  difficult  to  determine 
exactly  why  response  rates  for  these 
enumerators  was  so  low  for  the  1999  survey 
year. 

CONCLUSION 

A  longitudinal  panel  approach  to  the 
ARMS  may  be  feasible  from  a  respondent 
cooperation  point  of  view.  During  the  two 
years  of  this  study  in  two  states,  we  were 
able  to  maintain  a  retention  rate  of  about  88 
percent,  and  did  not  experience  much 
negative  feedback.  However,  even  with 
increased  public  relations  and  promotion,  we 
should  expect  some  decrease  in  response 
rates  each  year,  as  well  as  lower  response 
rates  on  the  panel  than  on  the  regular  ARMS 
sample. 

In  addition,  the  promotional  work  we 
do  with  enumerators  may  wear  off  after  an 
initial  increase  in  motivation  and  increased 
response  rates.  This  is  seen  with  the 
increased  response  rates  in  the  year  that  we 
increased  the  promotional  work  and  the 
decrease  during  the  following  year. 

There  are  two  critical  questions  that 
still  need  to  be  answered  regarding  whether 
a  Panel  Plus  approach  is  feasible  for  the 
ARMS.  First,  can  a  short  questionnaire  be 
developed  for  use  with  the  Plus  sample  and 
will  response  rates  on  it  indeed  be  higher 
than  for  the  longer  version  of  the 
questionnaire  currently  used?  Secondly,  can 
the  Panel  and  the  Plus  be  combined  to 
satisfy  data  needs  and  provide  data  that  are 
reasonably  equivalent  to  what  are  produced 
now?  Until  these  questions  are  answered, 
we  cannot  accurately  assess  the  feasibility  of 
using  a  panel  sample  and  data  collection 
methodology  for  the  ARMS. 

In  general,  the  idea  of  a  three  year 
study  did  not  drastically  decrease  response 


rates  and  did  not  receive  a  high  level  of 
negative  respondent  reaction  in  Virginia  or 
Missouri.  Therefore,  it  may  be  feasible  to 
obtain  cooperation  for  a  three  year  period  for 
this  survey.  However,  a  larger  study  may 
need  to  be  done  to  assess  exactly  what 
impact  a  longitudinal  design  will  have  on 
response  rates  and  data  quality. 

RECOMMENDATIONS 

1.  Ongoing  public  relations  and 
promotion  of  NASS  and  the  ARMS  should 
be  increased  with  both  respondents  and 
enumerators  in  an  attempt  to  positively 
impact  response  rates  over  the  long  term. 

PR  seems  to  at  least  temporarily  increase 
response  rates,  so  an  ongoing  PR  program 
geared  towards  respondents  as  well  as 
enumerators  may  be  effective  over  a  long 
period  of  time. 

2.  When  a  short  questionnaire  has  been 
developed  and  formal  consideration  has 
been  given  to  the  panel  plus  sampling, 
modeling,  and  analysis,  this  portion  of  the 
ARMS  pilot  study  could  be  continued.  Until 
then,  this  public  relations  and  promotion 
aspect  of  the  pilot  study  should  be 
discontinued.  In  the  meantime,  panel 
operations  should  be  contacted  and  told  the 
study  will  not  continue,  since  many  were 
told  they  would  be  interviewed  for  three 
consecutive  years.  The  state  office  or 
interviewers  should  thank  them  for  their 
help  in  past  years  and  inform  them  that  the 
study  will  not  resume  in  February  2001. 

3.  To  seriously  consider  using  a  panel 
data  collection  methodology,  resources  need 
to  be  devoted  to  developing  a  short 
questionnaire,  a  panel  plus  sample  design 
and  estimation  methodology.  Only  when  all 
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of  these  components  are  looked  at  together 
can  the  feasibility  of  using  a  panel  be 
assessed. 
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Appendix  A 


Further  Discussion  of  Second  Year  Response  Rates 

As  stated  earlier,  in  Virginia,  two  of  the  first  cohort  enumerators  were  used  to  complete  some  of 
second  cohort  interviews.  Their  response  rates  were  not  included  in  the  graphs  shown  in  the 
body  of  this  report.  These  enumerators  were  used  because  it  was  difficult  for  the  Virginia  SSO  to 
always  match  counties  with  pilot  enumerators. 

For  informational  purposes,  the  response  rates  for  the  first  group  of  Virginia  enumerators  (two 
enumerators  that  interviewed  first  year  operations  during  both  the  first  and  second  years  of  the 
study)  are  compared  to  the  second  group  of  Virginia  enumerators  (three  enumerators  who  only 
interviewed  during  the  second  year)  below. 
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As  can  be  seen  in  the  above  graph,  the  first  group  of  enumerators  have  higher  response 
rates  for  their  1998  nonpanel  cases  and  their  1999  panel  cases.  However,  the  second  group  of 
enumerators  have  higher  response  rates  for  the  1999  nonpanel  cases  (however,  there  were  only  7 
nonpanel  cases  for  the  first  group  of  enumerators  for  this  category  because  of  all  the  returning 
and  new  panel  cases  they  were  interviewing  that  year).  Because  of  the  large  differences  between 
the  response  rates  for  the  first  and  second  groups  of  enumerators,  only  the  rates  for  the  second 
group  of  enumerators  were  presented  in  this  paper. 


Appendix  B 


Response  and  Refusal  Rate  Graphs  for  First  Year  Enumerators 
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Appendix  C 


Response  and  Refusal  Rate  Graphs  for  Second  Year  Enumerators 
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